<html>
<head><meta charset="utf-8"><title>prometheus counter over restart · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html">prometheus counter over restart</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="239587088"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239587088" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239587088">(May 20 2021 at 13:04)</a>:</h4>
<p><a href="/user_uploads/4715/2et_Qmk0Eg8Uq1GlTXNNPIE3/image.png">image.png</a> </p>
<div class="message_inline_image"><a href="/user_uploads/4715/2et_Qmk0Eg8Uq1GlTXNNPIE3/image.png" title="image.png"><img src="/user_uploads/4715/2et_Qmk0Eg8Uq1GlTXNNPIE3/image.png"></a></div><p>My query is <code>sum(playground_request_duration_seconds_count{edition != ""}) by (edition)</code>. I thought that Prometheus would automatically handle counter resets and increment the value. However, the server restart last night seems to have reset the graph. What piece am I missing?</p>



<a name="239591672"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239591672" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239591672">(May 20 2021 at 13:33)</a>:</h4>
<p>For now I've changed to a rate view, but I'd like the cumulative view as well, eventually.</p>



<a name="239593310"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239593310" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239593310">(May 20 2021 at 13:43)</a>:</h4>
<p>For certain functions that care about counter vs. gauge prometheus documentation for the operator will specify it like such:</p>
<blockquote>
<p>delta should only be used with gauges.</p>
</blockquote>
<p>I'm not sure what your query is trying to achieve though, but I think what is happening in your case is that after the reset your scale is way lower than before the reset.</p>



<a name="239593486"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239593486" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239593486">(May 20 2021 at 13:44)</a>:</h4>
<p>(i.e. you're summing the count, and so right after the restart your graph would show values in orders of 1000s and before the restart probably in millions)</p>



<a name="239593652"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239593652" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239593652">(May 20 2021 at 13:45)</a>:</h4>
<p>In this case I'd say that perhaps you may want a different kind of visualization if the goal is to present the ratio between different editions being used.</p>



<a name="239593931"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239593931" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239593931">(May 20 2021 at 13:47)</a>:</h4>
<p><code>sum_over_time</code> or <code>increase</code> may help too?</p>



<a name="239594114"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239594114" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239594114">(May 20 2021 at 13:48)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116155">Jake Goulding</span> <a href="#narrow/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart/near/239587088">said</a>:</p>
<blockquote>
<p><a href="/user_uploads/4715/2et_Qmk0Eg8Uq1GlTXNNPIE3/image.png">image.png</a> </p>
<p>My query is <code>sum(playground_request_duration_seconds_count{edition != ""}) by (edition)</code>. I thought that Prometheus would automatically handle counter resets and increment the value. However, the server restart last night seems to have reset the graph. What piece am I missing?</p>
</blockquote>
<p>my understanding is that prometheus itself does not handle counter resets on the underlying storage, it's up to the functions you're using to handle it on their own</p>



<a name="239595714"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239595714" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239595714">(May 20 2021 at 13:59)</a>:</h4>
<p><span class="user-mention silent" data-user-id="123586">nagisa</span> <a href="#narrow/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart/near/239593652">said</a>:</p>
<blockquote>
<p>In this case I'd say that perhaps you may want a different kind of visualization if the goal is to present the ratio between different editions being used.</p>
</blockquote>
<p>yeah, <span class="user-mention" data-user-id="116155">@Jake Goulding</span> on this I'd use an <code>increase</code> over an hour or so</p>



<a name="239595752"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239595752" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239595752">(May 20 2021 at 13:59)</a>:</h4>
<p>increase over 1 hour vs per-second rate<br>
<a href="/user_uploads/4715/e6sYMD-R2-2C6uxW_GdwTmAQ/2021-05-20-15-58-58.png">2021-05-20-15-58-58.png</a></p>
<div class="message_inline_image"><a href="/user_uploads/4715/e6sYMD-R2-2C6uxW_GdwTmAQ/2021-05-20-15-58-58.png" title="2021-05-20-15-58-58.png"><img src="/user_uploads/4715/e6sYMD-R2-2C6uxW_GdwTmAQ/2021-05-20-15-58-58.png"></a></div>



<a name="239595892"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239595892" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239595892">(May 20 2021 at 14:00)</a>:</h4>
<p>IMO we don't get that many requests making per-second rate useful</p>



<a name="239596135"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596135" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596135">(May 20 2021 at 14:01)</a>:</h4>
<p>What bothers me about the rates is the nonsense decimals. I'd rather it show integers.</p>



<a name="239596392"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596392" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596392">(May 20 2021 at 14:02)</a>:</h4>
<p>I mean, rate is accurate</p>



<a name="239596411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596411">(May 20 2021 at 14:02)</a>:</h4>
<p>it's the number of requests every second</p>



<a name="239596432"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596432" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596432">(May 20 2021 at 14:02)</a>:</h4>
<p>I didn't claim it was wrong.</p>



<a name="239596582"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596582" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596582">(May 20 2021 at 14:03)</a>:</h4>
<p><span aria-label="smiley" class="emoji emoji-1f603" role="img" title="smiley">:smiley:</span> yeah, for play's scale we should definitely use either <code>increase(metric[1m])</code> or <code>increase(metric[1h])</code></p>



<a name="239596709"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596709" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596709">(May 20 2021 at 14:04)</a>:</h4>
<p><span class="user-mention silent" data-user-id="123586">nagisa</span> <a href="#narrow/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart/near/239593652">said</a>:</p>
<blockquote>
<p>In this case I'd say that perhaps you may want a different kind of visualization if the goal is to present the ratio between different editions being used.</p>
</blockquote>
<p>This may indeed be the ticket. I'm not 100% sure what I want to see, even.</p>



<a name="239596724"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596724" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596724">(May 20 2021 at 14:04)</a>:</h4>
<p>the screenshot I posted uses 1h to have more "smooth" graphs, 1m has integers but it's still a lot noisy</p>



<a name="239596825"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239596825" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239596825">(May 20 2021 at 14:04)</a>:</h4>
<p>Looking forward to that Grafana update to get <code>$rate__interval</code> or whatever</p>



<a name="239597173"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239597173" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239597173">(May 20 2021 at 14:06)</a>:</h4>
<p>I think <code>$rate__interval</code> will still give you decimal places in rates</p>



<a name="239597235"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239597235" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239597235">(May 20 2021 at 14:07)</a>:</h4>
<p>sure, but it helps with losing the graph when zooming</p>



<a name="239621566"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239621566" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239621566">(May 20 2021 at 16:35)</a>:</h4>
<p><code>$__rate_interval</code>. If there's a desire to update grafana in near future for this, I would recommend doing so sooner rather than later – that same update introduces new implementations of certain panels, which tend to change the look&amp;feel of the dashboards significantly and don't mesh well with the old panels.</p>



<a name="239621713"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239621713" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> nagisa <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239621713">(May 20 2021 at 16:36)</a>:</h4>
<p>There's also a significant change to how panels are composed I think, which for somebody used to the old thing takes some getting used to.</p>



<a name="239639478"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239639478" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239639478">(May 20 2021 at 18:37)</a>:</h4>
<p>Yeah, I'm using some of the new panels locally to give "big number" overviews, which is nice.</p>



<a name="239639575"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/prometheus%20counter%20over%20restart/near/239639575" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/prometheus.20counter.20over.20restart.html#239639575">(May 20 2021 at 18:38)</a>:</h4>
<p>Just gotta wait quietly for the update to be scheduled.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>